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Abstract. Multi-sensor remote sensing data can significantly improve the interpretation and usage 
of large volume data sources. A combination of satellite Synthetic Aperture Radar (SAR) data and 
optical sensors enables the use of complementary features of the same image. In this paper, SAR 
data is injected into optical image using a combining fusion method based on the integration of 
wavelet Transform and IHS (intensity, Hue, and Saturation) transform. Not only to preserve the 
spectral information of the original (MS) image, but also to maintain the spatial content of the high- 
resolution SAR image. Two data sets are used to evaluate the proposed fusion algorithm: one of 
them is Pleiades, Turkey and the other one is Boulder, Colorado, USA. The different fused outputs 
are compared using different image quality indices. Visual and statistical assessment of the fused 
outputs displays that the proposed approach has an effective translation from SAR to the optical 
image. Hence, enhances the SAR image interpretability. 


1. Introduction 


Over the past decade, the fusion process that combine a complementary information from multi- 
source of data into a single resultant image such as optical remote sensing (RS) data as a passive 
sensor image or a very high spatial resolution microwave data as an active one has a number of 
additional benefits. This different data from platforms has become increasingly available to extract 
the urban features and land cover information from such data. It is well known that optical sensors 
can provide information about the reflective characteristics of features of the Earth surface, while 
the synthetic aperture radar (SAR) data measures the physical properties of texture and information 
on the surface roughness [1]. They can operate all day and night in all weather conditions unlike the 
optical sensors. Also, SAR sensor can acquire images that independent of the conditions of 
illumination [2]. However, SAR images of complex scenes such as urban areas cannot be visually 
interpreted in an accurate manner due to some radar imaging effects like the multiple-bounce 
propagation and speckle noise phenomenon that leads to imprecise object borders [3]. On the other 
hand, Optical images are much easier to interpret by human operators which represent the 
reflectance of solar energy from objects in a target area; therefore, they usually provide more details 
at similar resolution [4]. Therefore, it is desirable for many tasks to create a fused product by 
merging high spatial resolution SAR with moderate spatial resolution MS images to better 
understanding of the objects in target areas [5-6]. 

The motivation behind data fusion is to generate an interpretation of the scene not obtainable with 
data from a single sensor, or to form a composite image whose quality is superior to any of the 
individual input images [7]. Panchromatic image has a high spatial resolution gives geometric 
details of a data and multispectral image provides the visual performance by combining the 
characteristics of the optical sensors to integrate the multiple images into a resultant image that 1s 
more suitable for the human visual purpose or computer processing applications [8-9]. 
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Multi-sensor remote sensing pixel level data fusion approaches have been discussed in several 
studies. Generally, these methods can be classified among two categories: component substitution 
(CS) which is considered as a spatial domain fusion method and multi-resolution analysis (MRA) as 
a Transform domain fusion method. The CS approach is based on projecting the MS image into 
another space and replacing one component of MS image with the high spatial PAN image [5]. Due 
to the spectral distortion of CS methods, Multi-scale decomposition (MSD) approach has been 
performed. The MRA methods include Laplacian pyramid and the wavelet transform are based on 
MSD of the high spatial resolution imagery to extract the spatial details and then transferring into 
the color image. Graphically, the entire multi-scale representation looks like a pyramid (pyramid 
decomposition), with the original image on the bottom and smaller images from every cycle stacked 
one on another. The Pyramid-based image fusion methods, including Laplacian pyramid transform, 
were all developed from Gaussian pyramid transform which has been modified and widely used in a 
number of image processing applications including the image fusion [10] [11]. Particularly, The 
MRA fusion approaches usually to provide superior performance in terms of reducing the spectral 
distortion compared to CS or spatial domain based methods [12] [13]. 

On the other hand PCA transformation can acquire higher spatial resolution but provides more 
serious distortion of spectral characteristics. Pyramid based fusion not produces spectral 
degradation but its performance reduces with decomposition level. On the other hand, Wavelet 
transform based fusion method provides high SNR as well as no spectral degradation for fused 
images but suffers from directional selectivity during fusion. The A`trous wavelet transformation 
can preserve the spatial data while it lacks a high spatial resolution in the results. Hybrid methods 
use the advantages of both the CS and MRA methods with a combination of them [14]. Metwalli et 
al. [15] studied hybrid fusion method based on PCA and high pass filter (HPF) for providing pan- 
sharpened image with superior spatial resolution and less spectral distortion. It is found that the 
developed method is significantly reduced the spectral distortion compared with the PCA, HPF, and 
Gram-Schmidt (GS) fusion methods. 

This article aims to inject the spatial information of the SAR image into the optical image, while 
maintaining its spectral properties and obtaining an easy interpretable SAR features. An integrated 
IHS and wavelet fusion approach is presented to make full use of the merits of the IHS and wavelet 
transform fusion methods. 


2. Wavelet Transform and IHS 


Generally, Wavelet transformation (WT) is a mathematical tool for signal processing, has now been 
widely used in the optical image fusion domain, 1.e., fusing high spatial resolution PAN images 
with high spectral resolution MS images. Many image fusion methods based on wavelet 
transformation have been investigated. Base on multi-scale decomposition (MSD), Discrete 
Wavelet Transform (DWT) can decompose a high spatial resolution PAN image into high 
frequency and low frequency coefficients. The high frequency component from PAN can then be 
injected into the low spatial resolution MS image, resulting in a PAN-sharpened MS image with 
high spatial details which is generated by implementing an inverse discrete wavelet transform 
(IDWT). This basis could be adopted for the fusion between different sources of images such as 
SAR and optical images [6]. A stand-alone wavelet image fusion method can well retain the color 
information of the MS image, but with visible artifacts in the fusion results [16]. 
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Figure 1. 2-D DWT: The input image is decomposed in rows and columns by using low-pass (h,,) and 
high-pass (gn) filters and subsequent down sampling at each level to get approximation and details 
coefficients [17]. 
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Figure 2. Multi-level Stationary wavelets transform. 


The (intensity—hue-saturation) IHS fusion method falls under spatial domain techniques that can 
preserve the spatial characteristics of the high spatial resolution image, but causes a kind of color 
distortion depends on the correspondence between the high spatial resolution image and the 
intensity component of the low spatial resolution MS image (SAR image is considerably different 
from the intensity image of a MS image; thus, it usually causes significant color distortion) [18]. 
The integration of wavelet and IHS methods could utilize the merits of individual methods and 
avoid the drawbacks of the methods [6]. 

The R, G and B bands of the MS image are separated then transformed into HIS color space, the 
intensity (I), hue (H) and saturation (S), by forward IHS transform. Before replacing the I 
component with PAN image, we match the histogram of the PAN image to the histogram of I 
component to reduce the color distortion. Performing the inverse transformation to obtain a high 
spatial resolution MS image, but the fusion image suffers a serious spectral distortion. The main 
steps are illustrated in Fig. 3, of the standard IHS fusion scheme [19-20]. 
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Figure 3. Image Fusion with the IHS method. 


3. Methodology 


This study aims to inject the spatial information of the SAR image into the optical image, while 
maintaining its spectral properties and obtaining an easy interpretable SAR features. An integrated 
IHS and wavelet fusion approach is presented to make full use of the merits of the IHS and wavelet 
transform fusion methods. The followed flowchart of the proposed method is illustrated in Fig 4. 
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Figure 4. Flowchart of the SAR-Optical proposed image fusion algorithm. 
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Generally, the IHS transform is utilized to combine the color information of the optical image with 
the spatial detail of SAR image. The wavelet transform is implemented to extract spatial detail from 
SAR image and produce a new intensity image. 


3.1. Data Preprocessing 
3.1.1. Co-Registration of Optical and SAR Images 


Image-to-image registration refers to the process of aligning images so that their details overlap to 
correct the nonlinear geometrical differences between the two images. It is achieved by rotating 
and/or translating the test image to align with the reference image, which is a crucial step, so it 1s an 
important part to match them geometrically. However, image matching and data co-registration, 
such as optical and SAR imagery, considered as a challenge in remote sensing data fusion, it is still 
an open field of research for many years [21]. In our algorithm, the SAR image is regarded as a 
reference image and the optical image is the test one. It is achieved by automatically finding the 
affine transformation which maps a point set from reference image into the same corresponding 
points in the other image. 


3.1.2. Speckle Reduction of the TerraSAR Image 


Removal of the Speckle noise is a substantial step in further studies. The analysis of the microwave 
images must be based on the speckle effects suppression techniques such as local region, gamma 
map, and lee-sigma filters [22-6]. In general, speckle in SAR images is supposed to be modeled as 
multiplicative noise, which can be reduced by multi-look processing or spatial filtering. Here, the 
enhanced Lee filter was selected, such that the filtered image could preserve the inherent 
characteristics of the SAR image while minimizing speckle noise [23]. 


3.2 Image Fusion Algorithm 


A set of experiments that have been carried out on the different datasets are described in the 
following subsection. 


// This program to Fused of High-Resolution SAR and Optical Imageries Based on a Wavelet 
transform and IHS Integrated Algorithm 
1. Read SAR image in img! 
2. Read MSI image in img2 
3. Image Co_registration between img! and img2 
// Noise reduction of SAR image 
4. Despeckle img] 
5. Read Red, green and blue components as R, G, B 
// RGB to IHS conversion 
6. Calculate Num = (R-G)+(R-B) 
7. Calculate Dum = sqrt((R-G).*%2+(R-B).*(G-B) 
8. Th=acos 0.5 (( Num/ Dum )+eps) 
9. H Component = Th/(2*p1) 
10. Calculate NUS = min(min(R,G),B) 
11. Calculate DUS = (R+G+B-+eps) 
12. S Component = 1-3.*NUS/DUS 
13. I Component = =(R+G+B)/3 
// Histogram matching between pan and intensity 
14. img1_new = histogram matching between img1 and I component 
// level2 pan decomposition 
15. Extract Approximation_P, horizontal_P, vertical_P and diagonal_P coefficients using 2D 
Swt filter 
// level2 intensity decomposition 
16. Extract Approximation_I, horizontal_I, vertical_I and diagonal_I coefficients using 2D 
Swt filter 
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// Fused Approximation 
17. fused_diagonal component = masking and addition diagonal_P and diagonal_I coefficients 
18. fused_horizontal component = masking and addition horizontal_P and horizontal_I 
coefficients 
19. fused_vertical component = masking and addition vertical_P and vertical_I coefficients 
20. fused_ approximation component = pixel based mean rule between approximation_P and 
approximation_I coefficients 
// Inverse wavelet transform (reconstruction) 
21. New_I = Iswt for fused_diagonal, fused_horizontal, fused_vertical and fused_ 
approximation components 
// IAS TO RGB CONVERSION 
22.1mg2_new = Inverse IHS of New_I and H, S Component 


4. Quality Assessment 


There are many image fusion approaches have been proposed to efficiently fuse the MS and SAR 
images and between PAN and MS imageries. However, the quality judgment of the fused image 
should be assessed in different remote sensing applications [24]. Several indices for assessing the 
image quality have been proposed so far, which can also be implemented to evaluate the fusion 
performance in terms of statistical indicators. Image assessment methods can be generally divided 
into two classes: qualitative (or subjective) methods and quantitative (or objective) methods [25]. 

In the proposed structure, the performance of the fused image depends on the visual comparison 
between the fused image and raw input images [24]. A quality indicator that measures spectral and 
spatial similarity between multispectral and fused images is mainly visual [26]. Peak Signal-to-Noise 
Ratio, Root Mean Square Error, the spectral correlation coefficient and the Entropy. 


5. Results and Discussion 


A Pleiades image of 21 May 2018 and a TerraSAR-X image of 27 May 2018 have been used. The 
Pleiades data have four multispectral bands (B1: 0.45—0.52 mm, B2: 0.52—0.60 mm, B3: 0.63—0.69 
mm and B4: 0.76—0.90 mm) and one panchromatic band (Pan: 0.45—0.9 mm). The spatial resolution 
is 0.5 m for the panchromatic image, while it is 2 m for the multispectral bands. While TerraSAR-X 
parameters are shown in table 1. 


Table 1. The characteristics of the TerraSAR-X data. 


GROUND RANGE 
POLARIZATION DETECTED (GRD) FREQUENCY 





Testing Data with different spatial resolutions are at somewhere urban region from as displayed in 
Fig. 5 and Fig. 6. 
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Figure 6. The Original Pleiades and TerraSAR-X images (a) Pan image, (b) MS image (R, G and B), 
(c) SAR image. 


The SAR and panchromatic images were used as a high-resolution sample data in two comparative 
experiments to verify the transferring of the spatial information and color information to be well 
fused, spectral and spatial quality metrics were adopted to assess the data fusion quality. 





(b) 
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(c) (d) 


Figure 7. Fusion results between SAR and MS images using different methods 
(a) Using DWT. (b) Using IHS. (c) Using DWT-IHS. (d) 2D multi-level SWT. 


In Fig. 7. The standalone wavelet fusion method can enhance the spectral information of the fusion 
result than the original MS data and a severe color distortion exists when compared with HIS fusion 
method. By using the proposed wavelet-IHS fusion method, the spatial information of the SAR data 
can be incorporated into the MS images and easily interpreted than those in the original SAR data 
and also clearer than the original MS image even though the correlations between the SAR and MS 
images are very low. From the above fusion results, it is easy to see that SWT fusion method 
preserve the spatial information of the High-resolution SAR image and the spectral information of 
the original optical image. 

According to the image interpretation of the before and after fusion, it can be seen that a building is 
difficult to identify in all of the MS image because it is represented by only a few pixels; it is also 
difficult to identify in the SAR image because its appearance is different from that in a traditional 
optical image we usually see. However, in the fused images of the proposed wavelet-IHS method, 
roads and buildings can be much easier interpreted because of the integration of spatial and color 
information. 

The statistical comparison of the fusion results can evaluate the performance of each image fusion 
method in our study which consists of band 1, band 2, and band 3 according to (BGR) spectral 
bands. Table 2 shows the spectral Quantitative comparison between the fused image bands and the 
corresponding bands of the original MS image for the stand-alone wavelet, standalone IHS and the 
proposed wavelet-IHS methods, respectively. In general, the proposed DWT-IHS method is 
significantly higher than those of the other three methods in terms of Spectral CC and E. The 
proposed wavelet-IHS fusion result exhibits the lowest values of MSE among three fusion method, 
which means the less color distortion and the closer to original MS image and generates the highest 
PSNR values. 

Generally, Multi-sensor data are not captured at the same shooting time. After fusing the SAR 
details into the MS images, the IHS and wavelet results have significantly poorer color information 
than the proposed method according to Table 3. The spatial CCs between all the fused images with 
the SAR are much higher than that of the SAR with the original MS image (R band: 0.013; G band: 
0.0194; and B band: 0.024); this indicates that the spatial information from SAR has been 
substantially injected into the MS image after the fusion. Nevertheless, The IHS fusion method and 
the wavelet fusion method are slightly higher than those by using the proposed wavelet-IHS method 
as shown in table 2. Overall, the fusion quality of the proposed method is obviously higher than 
those of the stand-alone IHS and wavelet fusion methods in terms of color preservation and spatial 
information. Also, In terms of interpretability as well. 
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Table 2. Quantitative results of different fusion approaches between SAR and MS images. 


Method 


are 0.700 76.43 | 75.64 | 74.90 | 0.0015 | 0.0017 | 0.002 6.89 
SWT (0.855 0.858 | 89.51 92.39 | 0.00007 | 0.00005 |0.00003 | 6.828 6.69 6.42 


Table 3. High frequency component correlation coefficients of different fusion approaches between 
SAR and MS images. 


SAR Spatial CC 
Method 
pw 





0.298 | 69.31 | 68.04 | 66.59 | 0.0076 | 0.0102 | 0.014 6.65 





os 
C msowr | os | eae | oas 
SWT 054 


6. Conclusions 


This paper proposed a new method to fuse high spatial resolution PAN and SAR images with 
moderate spatial resolution MS image for smoothing the interpretation of SAR images. The new 
fusion method is based on the integration of DWT and IHS transform, because the standalone IHS 
and the wavelet fusion methods do not introduce satisfactory fusion results, especially at SAR and 
optical fusion, due to significant grey value differences (low CCs) between SAR images and MS 
images. The integrated wavelet-IHS fusion method utilizes the merit of the IHS fusion in integrating 
spatial resolution information and the merit of the wavelet fusion in preserving color information. 
Visual evaluation presents that the color of the fusion result by the proposed SWT method is much 
closer to original MS image than other methods. In addition to, the statistical analysis shows that the 
spectral quality of the fusion results of the proposed SWT method is very well; whereas the color 
information of the stand-alone HIS, wavelet and the integrated wavelet-I[HS fusion results is 
substantially distorted. In terms of spatial detail, the fusion results by the improved method are 
higher than to those results obtained by the other individual methods. 

From these experiments that the performance of SWT appears to have a superior performance in 
terms of all of the image quality indexes than other traditional methods. The values of the evaluation 
parameters also lend support these comments. However, the image fusion based method has some 
limitations, and it requires a reference optical image covering the same scene with SAR image, As a 
continuation of this research, we note, from the related work survey, that image fusion continues to 
be an active and productive area of research. 
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